Internet Archive Web Crawls

The Internet Archive discovers and captures web pages through many different web crawls.

More...

RSS

ABOUT

COLLECTION

SHOW DETAILS

SORT BY

VIEWS

TITLE

DATE ARCHIVED

CREATOR

Prior Page

58.2M 58M

nsdlweb

ITEMS

58.2M

VIEWS

Jul 12, 2012 07/12

eye 58.2M

this data is currently not publicly accessible.

Live Web Proxy Crawls

812,009 812K

liveweb-20190323183335

Mar 23, 2019 03/19

eye 812,009

favorite 0

comment 0

49.2M 49M

Collections news crawls v2

820

ITEMS

49.2M

VIEWS

Dec 13, 2012 12/12

by ximm@archive.org

eye 49.2M

Live Web Proxy Crawls

1.5M 1.5M

liveweb-20170830174854

Aug 30, 2017 08/17

eye 1.5M

favorite 0

comment 0

Live Web Proxy Crawls

1.6M 1.6M

liveweb-20170830171650

Aug 30, 2017 08/17

eye 1.6M

favorite 0

comment 0

26.5M 27M

Edu & Gov Crawl, June 2010

704

ITEMS

26.5M

VIEWS

Jun 21, 2010 06/10

eye 26.5M

TEST COLLECTION: Crawl of .edu and .gov sites started in June 2010.
Topic: crawldata

41.6M 42M

End of Term 2012 Web Crawls

2,383

ITEMS

41.6M

VIEWS

Sep 13, 2012 09/12

eye 41.6M

This collection contains web crawls performed on the US Federal Executive, Legislative & Judicial branches of government in 2012-2013.
Topics: end of term, US, Federal government, 2012, Obama

Live Web Proxy Crawls

637,519 638K

liveweb-20170607234704

Jun 7, 2017 06/17

eye 637,519

favorite 0

comment 0

Live Web Proxy Crawls

722,125 722K

liveweb-20170904014127

Sep 4, 2017 09/17

eye 722,125

favorite 0

comment 0

Live Web Proxy Crawls

587,385 587K

liveweb-20170820233510

Aug 20, 2017 08/17

eye 587,385

favorite 0

comment 0

41.5M 41M

ORG Survey Crawls

193

ITEMS

41.5M

VIEWS

Jun 20, 2011 06/11

eye 41.5M

Survey of .org domains. This data is currently not publicly accessible.

Live Web Proxy Crawls

565,648 566K

liveweb-20170626143957

Jun 26, 2017 06/17

eye 565,648

favorite 0

comment 0

Live Web Proxy Crawls

487,851 488K

liveweb-20181226102606

Dec 26, 2018 12/18

eye 487,851

favorite 0

comment 0

Live Web Proxy Crawls

502,165 502K

liveweb-20181226091254

Dec 26, 2018 12/18

eye 502,165

favorite 0

comment 0

Live Web Proxy Crawls

423,846 424K

liveweb-20170531232252

May 31, 2017 05/17

eye 423,846

favorite 0

comment 0

Live Web Proxy Crawls

10.7M 11M

liveweb-20190114034752

Jan 14, 2019 01/19

eye 10.7M

favorite 0

comment 0

Live Web Proxy Crawls

587,312 587K

liveweb-20170610172846

Jun 10, 2017 06/17

eye 587,312

favorite 0

comment 0

45.6M 46M

International News Crawl started in September 2010

1,083

ITEMS

45.6M

VIEWS

Oct 4, 2010 10/10

eye 45.6M

Crawl of International News Sites with initial seedlist and crawler configuration from Sep 1, 2010.

Live Web Proxy Crawls

415,785 416K

liveweb-20181227083118

Dec 27, 2018 12/18

eye 415,785

favorite 0

comment 0

Live Web Proxy Crawls

569,055 569K

liveweb-20170610224951

Jun 10, 2017 06/17

eye 569,055

favorite 0

comment 0

Live Web Proxy Crawls

539,289 539K

liveweb-20170611021416

Jun 11, 2017 06/17

eye 539,289

favorite 0

comment 0

21.5M 22M

Government Web & Data Archive

6,269

ITEMS

21.5M

VIEWS

Apr 6, 2017 04/17

eye 21.5M

This collaborative project is an extension of the 2016  End of Term  project, intended to document the federal government's web presence by archiving government websites and data. As part of this preservation effort, URLs supplied from partner institutions, as well as nominated by the public, will be crawled regularly to provide an on-going view of federal agencies' web and social media presence. Key partners on this effort are the Environmental Data & Governance...
Topics: government, data, federal, congress

28.1M 28M

survey_net00001

170

ITEMS

28.1M

VIEWS

Dec 29, 2011 12/11

eye 28.1M

Survey crawl of .net domains started October 2011.
Topics: webwidecrawl, net

17M 17M

End Of Term 2016 UNT Crawls

1,275

ITEMS

17M

VIEWS

Aug 28, 2017 08/17

eye 17M

End of Term 2016 Web Archive government web crawls by project partner the University of North Texas.
Topics: end of term, federal government, 2016, president, congress, university of north texas

Live Web Proxy Crawls

4M 4.0M

liveweb-20200521082552

May 21, 2020 05/20

eye 4M

favorite 0

comment 0

Live Web Proxy Crawls

429,904 430K

liveweb-20190417173954

Apr 17, 2019 04/19

eye 429,904

favorite 0

comment 0

Wide Crawl Number 16: Started June 3rd, 2017 - Still running

303,187 303K

Webwide Crawldata 2017-10-15T11:28:54PDT to 2017-10-15T06:00:00PDT

Oct 15, 2017 10/17

by Internet Archive

eye 303,187

favorite 0

comment 0

Internet Archive crawldata from Webwide Crawl, captured by crawl426.us.archive.org:wide from Sun Oct 15 11:28:54 PDT 2017 to Sun Oct 15 06:00:00 PDT 2017.
Topic: crawldata

Live Web Proxy Crawls

419,467 419K

liveweb-20190417175025

Apr 17, 2019 04/19

eye 419,467

favorite 0

comment 0

Live Web Proxy Crawls

415,254 415K

liveweb-20190417174511

Apr 17, 2019 04/19

eye 415,254

favorite 0

comment 0

24.3M 24M

YouTube 2007 Crawl

73,847

ITEMS

24.3M

VIEWS

Jun 12, 2012 06/12

eye 24.3M

Data crawled from YouTube.com in 2007 by Internet Archive. These files are not currently accessible.

23.7M 24M

Crawl Data

32,980

ITEMS

23.7M

VIEWS

Jun 20, 2011 06/11

eye 23.7M

Crawl Data. This data is currently not publicly accessible.

23.7M 24M

crawl_UNK

32,949

ITEMS

23.7M

VIEWS

Jan 26, 2011 01/11

eye 23.7M

Crawl data. This data is currently not publicly accessible.

Live Web Proxy Crawls

423,004 423K

liveweb-20190417175524

Apr 17, 2019 04/19

eye 423,004

favorite 0

comment 0

Live Web Proxy Crawls

365,297 365K

liveweb-20170608190202

Jun 8, 2017 06/17

eye 365,297

favorite 0

comment 0

Live Web Proxy Crawls

306,040 306K

liveweb-20181225160511

Dec 25, 2018 12/18

eye 306,040

favorite 0

comment 0

19.1M 19M

Wide Crawl started September 2010

332

ITEMS

19.1M

VIEWS

Oct 5, 2010 10/10

eye 19.1M

Web wide crawl with initial seedlist and crawler configuration from September 2010

Wide Crawl Number 18

400,922 401K

Wide18 Crawl Data 2021-08-12T22:36:46PDT to 2021-08-12T16:30:58PDT

Aug 13, 2021 08/21

by Internet Archive

eye 400,922

favorite 0

comment 0

Internet Archive crawl data from the wide crawl number 18, captured by crawl808.us.archive.org:wide18 from Thu Aug 12 22:36:46 PDT 2021 to Thu Aug 12 16:30:58 PDT 2021.
Topic: crawldata

21.1M 21M

End of Term 2008 California Digital Library Crawl

415

ITEMS

21.1M

VIEWS

Apr 3, 2012 04/12

eye 21.1M

End of term 2008 crawl data gathered by Internet Archive on behalf of the California Digital Library. This data is currently not publicly accessible.

20.3M 20M

Shallow Crawl Started February 2013

ITEMS

20.3M

VIEWS

Feb 19, 2013 02/13

eye 20.3M

Shallow crawl started February 2013 that collects content 1 level deep, including embeds. Access to content is restricted. Please visit the Wayback Machine to explore archived web sites.

Live Web Proxy Crawls

213,151 213K

liveweb-20170530010903

May 30, 2017 05/17

eye 213,151

favorite 0

comment 0

Live Web Proxy Crawls

987,413 987K

liveweb-20170831075218

Aug 31, 2017 08/17

eye 987,413

favorite 0

comment 0

Live Web Proxy Crawls

197,869 198K

liveweb-20170530013616

May 30, 2017 05/17

eye 197,869

favorite 0

comment 0

Live Web Proxy Crawls

226,360 226K

liveweb-20170530005031

May 30, 2017 05/17

eye 226,360

favorite 0

comment 0

Live Web Proxy Crawls

665,882 666K

liveweb-20170902084751

Sep 2, 2017 09/17

eye 665,882

favorite 0

comment 0

Live Web Proxy Crawls

22.3M 22M

Liveweb Capture 2011-03-27T22:10:09PDT to 2011-03-28T05:27:05PDT

May 3, 2011 05/11

by Internet Archive

eye 22.3M

favorite 8

comment 1

Internet Archive Liveweb Capture from WaybackMachine, captured by wwwb-proxy0.us.archive.org:wbm from Sun Mar 27 22:10:09 PDT 2011 to Mon Mar 28 05:27:05 PDT 2011.
( 1 reviews )
Topic: crawldata

16.4M 16M

2004 Election

178

ITEMS

16.4M

VIEWS

Apr 4, 2012 04/12

eye 16.4M

2004 Election crawl performed by Internet Archive. This data is currently not publicly accessible.

16.7M 17M

World Wars Crawl

ITEMS

16.7M

VIEWS

Apr 5, 2012 04/12

eye 16.7M

Web data related to World Wars I and II collected by Internet Archive in an experimental crawl sponsored by National Endowment for the Humanities and JISC. This data is currently not publicly accessible.

Live Web Proxy Crawls

972,514 973K

liveweb-20170903082135

Sep 3, 2017 09/17

eye 972,514

favorite 0

comment 0

Live Web Proxy Crawls

3.9M 3.9M

liveweb-20200618182719

Jun 18, 2020 06/20

eye 3.9M

favorite 0

comment 0

Live Web Proxy Crawls

6.1M 6.1M

liveweb-20190526060013

May 26, 2019 05/19

eye 6.1M

favorite 0

comment 0

Live Web Proxy Crawls

743,435 743K

liveweb-20170903082612

Sep 3, 2017 09/17

eye 743,435

favorite 0

comment 0

Live Web Proxy Crawls

281,872 282K

liveweb-20190404192533

Apr 4, 2019 04/19

eye 281,872

favorite 0

comment 0

Live Web Proxy Crawls

1.7M 1.7M

Liveweb Capture 2012-06-04T16:05:30 UTC to 2012-06-05T00:00:01 UTC

Jun 5, 2012 06/12

by Internet Archive

eye 1.7M

favorite 0

comment 0

Internet Archive Liveweb Capture from WayBack Machine, captured by wwwb-liveweb.us.archive.org from 2012-06-04T16:05:30 UTC to 2012-06-05T00:00:01 UTC.
Topic: crawldata

Live Web Proxy Crawls

336,859 337K

liveweb-20170806233157

Aug 6, 2017 08/17

eye 336,859

favorite 0

comment 0

Live Web Proxy Crawls

333,304 333K

liveweb-20170807004001

Aug 7, 2017 08/17

eye 333,304

favorite 0

comment 0

13.1M 13M

Shallow Crawl Started November 2012

137

ITEMS

13.1M

VIEWS

Nov 17, 2012 11/12

eye 13.1M

Shallow crawl started November 2012 that collects content 1 level deep, including embeds. This data is currently not publicly accessible.

Live Web Proxy Crawls

3.2M 3.2M

liveweb-20170830064242

Aug 30, 2017 08/17

eye 3.2M

favorite 0

comment 0

12.5M 13M

Hurricane Katrina

112

ITEMS

12.5M

VIEWS

Apr 6, 2012 04/12

eye 12.5M

Data related to Hurricane Katrina collected in 2005 by Internet Archive. This data is currently not publicly accessible. from Wikipedia : Hurricane Katrina was the deadliest and most destructive Atlantic hurricane of the 2005 Atlantic hurricane season. It was the costliest natural disaster, as well as one of the five deadliest hurricanes, in the history of the United States. Among recorded Atlantic hurricanes, it was the sixth strongest overall. At least 1,833 people died in the hurricane and...

Live Web Proxy Crawls

1M 1.0M

liveweb-20170903083212

Sep 3, 2017 09/17

eye 1M

favorite 0

comment 0

Wide Crawl Number 15: Started Oct 1st, 2016 - Ended May 8th, 2017

6.2M 6.2M

Webwide Crawldata 2016-10-16T02:25:38PDT to 2016-10-15T21:19:02PDT

Oct 16, 2016 10/16

by Internet Archive

eye 6.2M

favorite 0

comment 0

Internet Archive crawldata from Webwide Crawl, captured by crawl427.us.archive.org:wide from Sun Oct 16 02:25:38 PDT 2016 to Sat Oct 15 21:19:02 PDT 2016.
Topic: crawldata

Live Web Proxy Crawls

4.4M 4.4M

liveweb-20190506075757

May 6, 2019 05/19

eye 4.4M

favorite 0

comment 0

Live Web Proxy Crawls

13.4M 13M

liveweb-20180617131556

Jun 17, 2018 06/18

eye 13.4M

favorite 0

comment 0

Live Web Proxy Crawls

246,830 247K

liveweb-20181225170338

Dec 25, 2018 12/18

eye 246,830

favorite 0

comment 0

Live Web Proxy Crawls

349,849 350K

liveweb-20190403184344

Apr 3, 2019 04/19

eye 349,849

favorite 0

comment 0

Wide Crawl Number 17: Started August 3rd, 2018

7.6M 7.6M

Webwide Crawldata 2019-05-13T17:55:38PDT to 2019-05-13T13:45:55PDT

May 13, 2019 05/19

by Internet Archive

eye 7.6M

favorite 3

comment 8

Internet Archive crawldata from Webwide Crawl, captured by crawl805.us.archive.org:wide from Mon May 13 17:55:38 PDT 2019 to Mon May 13 13:45:55 PDT 2019.
( 8 reviews )
Topic: crawldata

Live Web Proxy Crawls

1.2M 1.2M

liveweb-20161030062141

Oct 30, 2016 10/16

eye 1.2M

favorite 0

comment 0

Live Web Proxy Crawls

998,452 998K

liveweb-20170903083005

Sep 3, 2017 09/17

eye 998,452

favorite 0

comment 0

12.6M 13M

End of Term 2008 UNT Dot Gov Crawl

282

ITEMS

12.6M

VIEWS

Apr 3, 2012 04/12

eye 12.6M

End of term 2008 crawl of .gov domains gathered by University of North Texas . This data is currently not publicly accessible. UNT is a student-focused, public, research university located in Denton, Texas.

Live Web Proxy Crawls

691,720 692K

liveweb-20170320201850

Mar 20, 2017 03/17

eye 691,720

favorite 0

comment 0

Live Web Proxy Crawls

234,963 235K

liveweb-20181225194727

Dec 25, 2018 12/18

eye 234,963

favorite 0

comment 0

Shallow Crawl Started 2013

453,703 454K

Shallow Webwide Crawldata 2013-06-09T02:57:23PDT to 2013-06-08T21:30:29PDT

Jun 9, 2013 06/13

by Internet Archive

eye 453,703

favorite 0

comment 0

Internet Archive crawldata from Shallow Webwide Crawl, captured by crawl455.us.archive.org:shallow from Sun Jun 9 02:57:23 PDT 2013 to Sat Jun 8 21:30:29 PDT 2013.
Topic: crawldata

Wide Crawl started February 2014

9.9M 9.9M

Webwide Crawldata 2014-02-19T01:09:37PST to 2014-02-18T21:33:27PST

Feb 19, 2014 02/14

by Internet Archive

eye 9.9M

favorite 0

comment 0

Internet Archive crawldata from Webwide Crawl, captured by crawl453.us.archive.org:wide from Wed Feb 19 01:09:37 PST 2014 to Tue Feb 18 21:33:27 PST 2014.
Topic: crawldata

Wide Crawl started February 2014

9.7M 9.7M

Webwide Crawldata 2014-02-19T07:58:38PST to 2014-02-19T05:13:46PST

Feb 19, 2014 02/14

by Internet Archive

eye 9.7M

favorite 0

comment 1

Internet Archive crawldata from Webwide Crawl, captured by crawl426.us.archive.org:wide from Wed Feb 19 07:58:38 PST 2014 to Wed Feb 19 05:13:46 PST 2014.
( 1 reviews )
Topic: crawldata

Live Web Proxy Crawls

900,064 900K

liveweb-20170903083828

Sep 3, 2017 09/17

eye 900,064

favorite 0

comment 0

Live Web Proxy Crawls

1.4M 1.4M

liveweb-20170903072754

Sep 3, 2017 09/17

eye 1.4M

favorite 0

comment 0

MORE RESULTS
Fetching more results

DESCRIPTION

The Internet Archive discovers and captures web pages through many different web crawls.

At any given time several distinct crawls are running, some for months, and some every day or longer.

View the web archive through the Wayback Machine.

ACTIVITY

comment

Collection Info

Access-restricted: true

Addeddate: 2010-06-11 18:34:15

Collection: web

Identifier: webwidecrawl

Mediatype: collection

Public-format: Metadata
Symlink Instructions
Collection Header
JPEG
JPEG Thumb
PNG
Animated GIF
Item Tile

Publicdate: 2010-06-11 18:34:36

Show_hidden_subcollections: true

Storage_size: 16.7 PB (in 63,179,589 files)

Subject: webwidecrawl

Title: Internet Archive Web Crawls

Created on

June 11
2010

ARossi
Archivist

ADDITIONAL CONTRIBUTORS

Wayback Machine Web Crawling
Archivist

VIEWS

Total Views 53,885,762,639 (Older Stats)

ITEMS

Total Items 1,814,201 (Older Stats)

TOP REGIONS (LAST 30 DAYS)

(data not available)

Internet Archive Audio

Featured

Top

Images

Featured

Top

Software

Featured

Top

Books

Featured

Top

Video

Featured

Top

Mobile Apps

Browser Extensions

Archive-It Subscription

Save Page Now

Internet Archive Web Crawls

Filters 1,815,014 RESULTS

Media TypeMedia Type

YearYear

Topics & SubjectsTopics & Subjects

CollectionCollection

CreatorCreator

LanguageLanguage

eye 58.2M

eye 812,009

favorite 0

comment 0

eye 49.2M

eye 1.5M

favorite 0

comment 0

eye 1.6M

favorite 0

comment 0

eye 26.5M

eye 41.6M

eye 637,519

favorite 0

comment 0

eye 722,125

favorite 0

comment 0

eye 587,385

favorite 0

comment 0

eye 41.5M

eye 565,648

favorite 0

comment 0

eye 487,851

favorite 0

comment 0

eye 502,165

favorite 0

comment 0

eye 423,846

favorite 0

comment 0

eye 10.7M

favorite 0

comment 0

eye 587,312

favorite 0

comment 0

eye 45.6M

eye 415,785

favorite 0

comment 0

eye 569,055

favorite 0

comment 0

eye 539,289

favorite 0

comment 0

eye 21.5M

eye 28.1M

1,815,014
RESULTS

Media Type

Year

Topics & Subjects

Collection

Creator

Language